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Abstract 

The  signature  tables  used  in  TAC  ASSESSOR  were  assumed 
to  be  representations  of  some  corresponding  value  functions 
evaluated  at  certain  levels  of  the  input  features.  The 
theoretical  mean-squared-error  due  to  this  substitution  was 
calculated  for  signature  table  1  and  table  5  in  the  FAC 
Target  Importance  Tables.  The  calculations  were  performed 
under  two  different  assumption  concerning  the  assumed  value 
functions. 

Case  1  was  that  the  signature  table  values  resulted  from 
evaluating  the  assumed  value  function  at  the  given  input 
feature  values.  Case  2  was  that  the  function  was  evaluated 
at  some  feature  values  in  the  given  quantization  level 
indicated  by  the  input  feature.  The  mean  squared  error  was 
then  expressed  as  a  percentage  of  the  assumed  mean  squared 
signal.  The  percent  errors  derived  were  20.481  and  27.811 
for  signature  table  1  under  Case  1  and  2  assumptions,  respectively. 

Next,  an  evaluation  simulation  was  performed  on  three  sets 
of  30  targets  whose  features  were  generated  using  uniform, 
exponential,  and  normal  random  variate  generators  scaled  to 
the  appropriate  feature  limits.  The  major  result  of  this 
was  that  a  significant  number  of  the  targets  were  involved 
in  ties  when  evaluated  using  the  signature  table  method.  The 
number  of  ties  ranged  from  13  out  of  30  up  to  28  out  of  30 
targets  involved  in  evaluation  ties. 
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Finally,  the  Spearman  rank  correlation  coefficient  was 
used  to  test  the  null  hypothesis  that  there  was  no  association 
between  the  evaluation  methods.  The  null  hypothesis  was 
rejected  in  every  case  at  the  0.99  confidence  level.  Based 
on  these  results,  it  is  the  recommendation  of  this  study  that 
formal  application  of  multi  attribute  utility  assessment  techniques 
be  applied  to  generate  a  substitute  for  the  signature  table 
evaluation  technique  applied  in  this  case. 
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I.  Introduction 


1  2 

*  The  question  of  Command  and  Control  (C  )  System  effective¬ 
ness  has  become  increasingly  important  to  the  military  estab¬ 
lishment.  Some  of  the  reasons  for  this  increased  importance 
include  the  increasing  value  of  limited  resources  and  the 

possible  consequences  of  wrong  decisions  due  to  inadequate, 

2 

inaccurrate,  and  erroneous  C  information.  As  resources 

become  more  scarce  and  valuable,  it  becomes  imperative  to 

increase  their  effective  employment.  As  the  magnitude  of  the 

possible  consequences  of  s  ginale  decision  increase,  so  must 

2 

the  amount  of  invalid  C  information  decrease. 

As  the  costs  of  research,  development,  procurement,  and 

2 

maintenance  of  elements  of  the  C  system  increase,  experimenta- 

♦  tion  with  modifications  and  new  equipment  has  a  tendency  to 

4 

decrease  in  effectiveness.  The  military  establishment  and, 

in  particular,  the  Air  Force,  has  moved  towards  system  simu- 

2 

lation  to  evaluation  proposed  changes  to  C  systems.  The  common 

goal  of  this  movement  is  to  gain  a  better  understanding  of 

2 

the  relationship  between  C  and  force  effectiveness. 

2 

A  major  stumbling  block  in  C  simulation  has  been  and 
continues  to  be  the  simulation  of  the  decision  making  process 
of  commanders  within  the  system.  One  alternate  approach  to 
simulating  the  decision  makers  is  to  employ  interactive  or 
"man  in  the  loop"  schemes.  There  are,  however,  some  serious 
drawbacks  to  this  approach.  The  first  is  the  time  and  cost 
involved  with  gathering  together  suitable  personnel  for  valid 
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system  operation.  Second,  is  the  time  required  to  execute 
completely  interactive  model.  Finally,  the  lack  of  consistency 
in  the  simulations  due  to  different  players  may  make  it 
difficult  to  verify  or  validate  the  results.  These 
deficiencies  have  generated  a  movement  towards  the  field  of 
artificial  intelligence  in  order  to  simulate  the  decision 
processes  necessary. 

Artificial  intelligence  techniques  attempt  to  capture  the 

decision  making  process  to  a  certain  degree  so  that  a  given 

input  will  yield  a  consistent  response  from  the  simulation. 

Signature  table  analysis  is  one  of  these  techniques  and  has 

2 

been  applied  in  C  system  modeling  as  a  means  of  simulating 
the  evaluation  of  situations,  alternatives,  and  targets. 
Research  Question 

The  application  of  signature  table  analysis  to  the  evalu¬ 
ation  of  some  class  of  objects  requires  that  the  class  of 
objects  be  characterized  in  terms  of  features.  These  features 
must  be  quantized  for  evaluation  by  the  signature  table  and 
this  quantization  process  generates  error  in  the  evaluation. 
What  error  results  from  the  application  of  signature  evalu¬ 
ation  techniques  to  objects  which  are  characterized  in  terms 
of  continuous  features  which  must  be  quantized? 

Methodology 

The  first  step  in  this  study  will  be  the  development  of 
a  theoretical  measure  of  error.  This  error  measure  will  be 
derived  by  assuming  that  a  signature  table  is  a  piecewise 
representation  of  a  continuous  value  function  derived  from  the 
decision  maker's  preferences  evaluated  over  the  given  features. 


This  theoretical  error  measure  will  be  converted  to  a  percentage 

error  measure  for  intuitive  appeal  and  will  be  applied  to  the 

2 

FAC  Target  Importance  Tables  used  in  the  TAC  ASSESSOR  C 
system  model  under  development  at  Air  Force  Studies  and 
Analysis.  Finally,  sample  target  data  sets  will  be  generated 
and  evaluated  using  both  the  signature  table  evaluation 
scheme  and  the  assumed  value  function  method.  The  targets  will 
be  rank  ordered  and  the  Spearman  rank  correlation  coefficient 
will  be  used  to  test  the  null  hypothesis  that  a  significant 
difference  exists  between  the  two  evaluation  schemes. 

Overview 

A  description  of  signature  table  evaluation  techniques  is 
presented  in  Chapter  II,  and  Chapter  III  contains  a  discussion 
of  how  these  techniques  are  applied  in  the  TAC  ASSESSOR  model. 
Chapter  IV  develops  the  error  measures  to  be  used  in  Chapter  V 
on  the  FAC  Target  Importance  Tables  used  in  TAC  ASSESSOR. 

The  last  chapter  contains  the  conclusions  and  recommendations 
of  the  study. 


II.  Signature  Tables 


Signature  tables  as  discussed  in  this  study  are  a  means 
of  evaluating  some  object  or  situation.  The  resultant 
evaluation  is  a  measure  of  worth  or  utility  that  represents 
the  decision  maker's  preference  for  the  combination  of 
features  that  represent  the  object  being  evaluated. 

The  first  step  in  the  application  of  signature  tables 
as  an  evaluation  tool  is  to  determine  a  set  of  features  that 
will  characterize  that  class  of  objects.  For  example,  a 
class  U  of  ICBM's  might  be  characterized  by  a  set  of  n 

.  .  ,  Fn>  where 

payload  capacity 
range 

reliability 

other  characteristics 


Each  object  to  be  evaluated  is  then  measured  in  terms  of  the 
features  to  be  used  in  the  evaluation.  These  measured  values 
are  then  quantized  for  use  by  the  signature  table  evaluation 
scheme. 


features  {F^  F2,  F^, 


F„  - 


« 
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Feature  Quantization 

Since  the  signature  table  can  accept  only  a  finite  number 
of  input  points,  it  must  be  decided  how  many  different 
values  each  feature  will  be  allowed  to  assume.  The  number 
of  quantization  levels,  q^,  determines  the  range  of  values 
that  feature  F^  can  assume.  \  The  number  of  quantization  levels 
will  depend  on  how  unique  each  object  is  when  measured  by 
the  i**1  feature.  There  must  also  be  determined  a  set  of 
threshold  values  which  serve  to  assign  the  measurements 
made  into  the  appropriate  quantization  level.  Finally,  the 
actual  values  which  will  represent  the  quantization  level  of 
a  particular  object  being  measured  have  to  be  determined.  The 
most  common  scheme  is  to  assign  the  middle  quantization  level 
the  value  0  and  assign  +1,  +2,  ...  to  quantization  levels 
above  and  -1,  -2,  .  .  .  to  quantization  levels  below. 

In  the  example,  suppose  the  largest  ICBM  payload  capacity 
in  the  class  U  is  10  Megatons  (10MT)  and  the  smallest  is 
SO  Kilotons  (50KT) .  Assume  for  this  example  that  only  three 
quantization  levels  are  needed  to  preserve  the  information 
contained  in  the  payload  capacity  feature.  One  explanation 
for  this  may  be  that  the  payloads  need  only  be  characterized 
in  terms  of  low,  medium,  or  high.  Another  could  be  that  the 
actual  payloads  naturally  divide  into  three  groups.  The 
threshold  values  may  be  derived  from  natural  divisions  in 
the  data  if  present  or  assigned  on  some  other  basis.  For 
this  example  let  the  first  threshold  value  be  500KT  and  the 
second  1MT.  Any  payload  less  than  500KT  will  be  assigned 
the  first  quantization  level.  A  payload  between  500KT  and 
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1MT  will  be  assigned  the  second,  and  the  third  level  will  be 

■m 

1  assigned  to  payloads  greater  than  1MT.  Finally,  quantized 

values  need  to  be  assigned  to  the  quantization  levels.  For 
the  purposes  of  this  example  let  -1,  0,  +1  be  assigned  to 
the  first,  second,  and  third  levels,  respectively.  The 
desired  result  is  that  each  feature  has  assigned  to  it  the 
smallest  possible  number  of  quantization  levels  in  order  to 
keep  the  size  of  the  signature  tables  manageable. 

Signature  Type  Formation 

The  next  step  in  signature  table  evaluation  is  to  group 
features  into  subsets.  These  subsets  of  features  are  known 
as  signature  types  (Ref  5:610).  No  generally  accepted  theory 
exists  on  the  method  of  grouping  the  features  into  signature 
+  types  and  the  question  that  presents  itself  is  whether  to 

group  highly  correlated  features  into  signature  types  or 
spread  these  correlated  features  among  the  set  of  signature 
types.  (Ref  3:77).  According  to  Page,  both  combinations 
can  yield  "excellent"  tables  (Ref  3:82). 

The  last  step  in  the  table  construction  is  the  determination 
of  a  derived  feature  for  each  possible  combination  of  values 
in  each  signature  type.  For  example,  there  are  three  features 
in  a  particular  signature  type  (Fj,  Fj,  F^}  and  each  of  these 
features  has  q^  quantization  levels,  then  there  are 
(<1*)  CqjHq^or  "(qj),  i  ■  1»3,4  combinations  of  values 
possible  in  that  signature  type.  Each  of  these  possible 
combinations  is  a  signature. 

(  Continuing  with  the  ICBM  evaluation  example,  let  features 

F^,  Fj,  and  Fj  be  grouped  into  one  such  signature  type. 

6 
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Suppose  the  quantization  levels  have  been  chosen  as  follows: 
qx  «  3  (-1,0, +1) 
q3  -  2  (0,+l) 
q4  =  3  (-1,0, +1) 

The  signature  table  for  this  signature  type  would  be 
constructed  with  the  tabular  headings  as  indicated  in  Figure  1. 

For  each  of  the  18  possible  signatures  a  derived  feature  value 
is  determined  which  represents  an  evaluation  of  each  signature. 
Derived  Feature  Determination 

Determination  of  the  value  of  the  derived  feature  for 
each  signature  can  be  accomplished  using  several  approaches. 

These  approaches,  however,  may  be  grouped  into  two  general 
strategies.  The  first  is  simply  a  direct  assessment  technique. 

The  analyst  or  data  user  determines  the  relative  merit  of  each 
signature  and  assigns  some  value  D  to  each  of  the  ?(q^)  “  k 
possible  signatures  for  each  signature  type.  The  other 
general  method  of  derived  feature  value  determination  involves 
the  use  of  a  computer  algorithm  to  make  the  assignment  based  on 
analysis  of  sample  data.  This  procedure  is  often  referred  to  as 
training.  In  order  to  train  a  signature  table, a  set  of  input  data  is 
generated.  Each  object  in  the  training  sample  is  measured  in 
terms  of  the  n  features  previously  selected  and  also  is  assigned 
a  key  feature,  K,  which  characterizes  the  object  in  general 
terms  such  as  good/bad  or  desirable/undesirable.  At  this  point, 
the  m**1  object  of  the  training  sample  can  be  described  by 
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Fig  1.  Example  Signature  Table  Organization 
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a  data  set,  S  ,  which  consists  of  values,  X.  ,  assigned  to 
m  1  ,m  ° 

each  feature,  F. ,  and  a  key  feature  value,  K  . 

S  *  {X.  ,  X-  ,  X-  ,  ...»  X  .  K  } 
m  lm  Zm  3m  '  nm  m 

All  data  sets  of  the  training  sample  comprise  the  input  to 
a  computer  algorithm  that  determines  the  value  of  D  for  each 
of  the  k  possible  signatures  in  each  signature  type.  Each 
object's  data  set  is  checked  for  the  values  of  the  features 
for  each  signature  type.  For  the  ICBM  example,  let  one 
ICBM's  data  set  be 

sn  -  10,  3,  1,  -1 . K  -  1) 

The  representative  signature  table  of  Figure  1  uses  a  signature 
type  composed  of  {F^,  Fj,  F^}  and  the  corresponding  values 
of  that  signature  type  for  the  ratl1  ICBM  taken  from  Sm 
would  be  {0,  +1,  -1}.  The  signature  table  is  searched 
for  this  particular  signature  and  a  count  is  made  of  the  key 
feature  values  of  that  signature.  The  derived  feature 
value,  D,  corresponding  to  that  signature  is  computed  based 
on  the  number  of  times  that  particular  signature  came  from  a 
training  object  with  a  good  key  feature  (K*l)  versus  the 
number  of  times  it  came  with  a  bad  key  feature  (Ks0).  Since 
each  signature  type  consists  of  only  a  few  of  the  features 
used  to  characterize  the  objects,  the  same  signature  could 
result  from  an  object  in  the  training  sample  with  either  a 
good  or  bad  key  feature. 

The  resulting  derived  features  from  each  signature  table 
comprise  a  new  derived  data  set  for  each  object.  These 
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derived  features  can  be  grouped  in  a  similar  manner  and 
higher  level  signature  tables  are  the  result.  The  final 
output  from  such  a  hierarchy  of  signature  tables  depends  on 
the  original  data  in  a  non-linear  way  (Ref  3:81). 

Current  Applications 

A.  L.  Samuel  has  done  considerable  work  in  this  area  by 
applying  signature  table  methods  to  a  checkers  playing  computer 
program.  He  characterized  moves  in  the  game  by  24  features 
which  described  the  resultant  board  situation.  The  24  features 
were  divided  into  six  signature  types ,  each  consisting  of 
four  features.  His  training  sample  consisted  of  a  library 
of  master  play  which  described  approximately  250,000  board 
situations.  The  master  library  games  were  played  by  the 
computer  during  training  and  each  possible  move  (object) 
was  evaluated  in  terms  of  the  24  features.  Each  move  was 
either  a  move  recommended  by  the  master  library  (good  key 
feature)  or  it  was  not  recommended  (bad  key  feature)  given 
the  current  board  situation.  The  signature  tables  were 
entered  for  each  signature  from  each  move  together  with  the 
key  feature  value.  Two  totals,  A  and  B,  were  tabulated  for 
each  of  the  possible  signatures  of  each  signature  type.  Each 
move  that  was  not  recommended  by  the  master  library  when 
entered  into  a  signature  table  caused  an  increase,  by  one, 
in  the  B  total  of  that  particular  signature.  A  move 
recommended  by  the  master  library  when  entered  into  a  signature 
table  caused  an  increase,  by  n,  in  the  A  total  of  that 
particular  signature,  where  n  was  the  number  of  non- 
recommended  moves  in  the  current  situation.  Samuel's  rationale 
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for  this  particular  scheme  was  to  place  more  importance  on 
the  master  library  recommended  moves  and  preserve  the  zero- 
sum  nature  of  the  game  (Ref  5:613).  Once  the  training  was 
complete,  values  for  the  derived  feature  of  each  signature 
were  determined  by  calculation  of  a  number  D  for  each 
signature,  where 


D  - 


A-B 

A+TT 


(1) 


If  the  output  of  this  table,  the  D  value,  was  the  final 
evaluation  in  a  hierarchy,  then  the  scheme  was  complete  and 
the  value  of  D  which  resulted  was  used  in  the  decision  making 
process.  If,  however,  this  signature  table  was  an  inter¬ 
mediate  table  in  a  hierarchy,  the  D  values  had  to  be  quantized 
for  input  to  the  next  table.  The  D  values  were  quantized  by 
first  rank  ordering  the  values  and  then  dividing  the  list  into 
the  desired  number  of  quantization  levels  and  quantized 
values  assigned  to  each  group  (Ref  5:613).  If  five  quan¬ 
tization  levels  wes  desired,  the  rank  ordered  D  values  would 
be  divided  into  five  equal  sized  groups.  The  signatures 
which  yielded  D  values  in  the  highest  group  would  be  assigned 
a  +2,  those  in  the  lowest  group  would  be  assigned  a  -2.  Values 
of  +1,  -1,  and  0  would  then  be  assigned  to  those  signatures 
which  yielded  calculated  D  values  which  fell  into  the 
second  highest,  second  lowest,  and  middle  groups,  respectively. 

C.  V.  Page  has  also  applied  the  concept  of  signature 
table  analysis.  He  used  the  technique  to  evaluate  housing 
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quality  in  Detroit,  Michigan,  on  the  basis  of  12  features. 

Some  of  the  12  features  he  used  were  rubble  per  vacant 
lot,  dwelling  units  per  house,  and  trash  per  dwelling  unit 
(Ref  3:78).  Organization  of  his  data  was  similar  to  Samuel's 
application;  however,  he  added  a  statistical  test  for  signi- 
ance  to  the  value  assessment  of  the  derived  feature  in 
each  table.  Page  assumed  that  an  estimate  of  the  key 
feat  ure  of  a  data  set  being  high  was  known  (p).  Next,  a 
confidence  interval  was  cnosen  such  that  C  =  1-ct.  Using  p 
and  C,  he  calculated  the  critical  values  N^  and  N2  from  the 
binomial  distribution  such  that: 

P(A  <  Nx-1)  <  j  £  P(A  <  Nx)  ,  (2) 

and 

P (A  <  N2-l)  <  l-£  <  P (A  <  N2)  (3) 

where  A  was  a  different  count  mechanism  than  it  was  in 
Samuel's  scheme.  Page  used  A  to  represent  the  number  of  times 
a  particular  signature  was  the  result  of  a  training  subject 
with  a  high  key  feature.  The  values  of  and  N2  were  the 
calculated  threshold  values  in  a  three  level  scheme.  The 
value  of  the  derived  feature  for  each  signature  was  assigned 
on  the  basis  of  A  and  its  relationship  to  the  threshold 
values.  If  A  >  N2,  D  for  that  signature  was  assigned  the 
high  value,  +1.  Low  value,  -1,  was  assigned  if  A  <  and 
0  was  assigned  if  <  A  £  N2>  Page  called  this  the  interval 
aethod.  It  resulted  in  the  assignment  of  +1  and  -1  to 
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"unusual"  events  and  0  to  events  which  could  be  explained  by 
4  chance  (Ref  3:80). 

Using  a  single  confidence  interval.  Page's  method  yielded 
three  values  that  the  derived  feature  could  be  assigned. 

Using  two  confidence  intervals,  the  number  of  values  available 
can  be  increased  to  five  (Ref  3:80).  In  this  case,  threshold 
values  N2>  Nj,  and  N4  would  be  calculated  such  that  the 
derived  feature,  D,  took  the  following  values  for  the  given 
conditions : 

(4) 

(5) 

(6) 

(7) 

(8) 


D  «  -2,  N1  >  A 
D  -  -1,  Nx  <  A  <  N2 
D  -  0,  N2  <  A  <  N3 
D  -  +1,  N3  <  A  <  N4 
D  -  +2,  N4  <  A 


Both  Page  and  Samuel  used  a  training  sample  in  order  to 
calculate  the  derived  feature  for  each  signature  in  each 
table.  From  these  derived  features,  second  level  tables 
were  generated.  The  derived  features  were  grouped  into 
signature  types  in  the  same  manner  as  those  generated  for 
the  first  level.  The  second  level  tables  yielded  second 
level  derived  features  which  could,  in  turn,  be  used  to 
generate  higher  level  tables.  Samuel  used,  among  others,  a 
three  level  hierarchy  with  24  entry  features  grouped  into 
six  signatures  types  at  the  first  level.  This  hierarchy 
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is  shown  in  Figure  2  (Ref  5:611).  The  organization  used  by 
Page  is  shown  in  Figure  3  (Ref  3:84).  Twelve  entry  features 
were  used  to  construct  66  first  level  tables  which  correspond 
to  all  possible  pairwise  combinations  of  the  entry  features. 

As  can  be  seen,  each  entry  level  feature  is  used  in  more  than 
one  signature  type  and  table.  These  redundancies,  with 
redundancies  in  the  grouping  of  higher  level  derived  features, 
yielded  a  six  level  hierarchy. 

The  end  result  of  the  direct  assessment  or  training 
methods  is  the  same:  A  hierarchical  structure  of  tables, 
each  consisting  of  a  list  of  possible  signatures  and  a 
resultant  derived  feature  value  for  each  signature.  Once 
the  derived  features  for  all  signatures  have  been  determined, 
evaluation  of  new  or  unassessed  objects  can  be  made.  The 
tables  now  act  as  a  predictor:  They  makes  inferences  about 
new  objects  submitted  for  analysis.  Given  a  new  signature, 
does  that  signature  come  from  an  object  that  is  good  or  bad? 
The  final  derived  feature  is  the  answer  to  that  question. 
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III.  Signature  Tables  in  TAC  ASSESSOR 


The  use  of  signature  tables  in  the  TAC  ASSESSOR  combined- 
arms  simulation  model  is  but  part  of  the  overall  effort  to 
simulate  the  decision  process  of  individual  commanders  and 
staffs  in  a  theater  level  conflict.  The  emphasis  is  on  the 
simulation  of  the  decision  maker  (DM)  who  distills  large 
sets  of  data  and  extracts  important  information  from  a  situation 
in  terms  of  combinations  of  features  which  characterize  the 
situation.  In  this  respect,  a  signature  table  "mimics  a 
skilled  analyst  who,  faced  with  the  task  of  analyzing  an 
extremely  large  and  complex  set  of  data,  is  forced  to  restrict 
his  attention  to  those  data  items  or  combinations  of  data 
items  that  are  in  some  sense  significant"  (Ref  6:61).  This 
chapter  will  focus  on  the  uses  of  signature  tables  in  TAC 
ASSESSOR  and  a  detailed  look  at  a  particular  hierarchy  of 
tables  used  in  the  ranking  of  targets  against  which  close 
air  support  (CAS)  sorties  are  assigned. 

Close  air  support  target  ranking  is  one  of  five  processes 
supported  through  the  use  of  signature  tables.  The  signature 
table  technique,  along  with  production  systems,  heuristics 
for  decision  tree  analysis,  and  directed  relational  graphs 
comprise  the  set  of  artificial  intelligence  techniques  used 
in  the  model.  Signature  table  techniques  are  applied 
specifically  to  the  processes  of  target  correlation,  battle 
assessment,  plan  evaluation  and  sortie  assignment  in  addition 
f  to  CAS  target  ranking  (Ref  6:66). 
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Target  correlation  is  the  process  of  comparing  new  information 
about  a  reported  unit  to  information  about  existing  units 
to  determine  if  the  new  information  applies  to  an  existing 
unit,  or,  if  it  implies  the  existence  of  a  unit  not 
previously  detected.  The  battle  assessment  application  of 
signature  tables  provides  evaluation  of  a  situation  in  terms 
of  estimated  combat  time  and  force  losses  and  is  used  to 
support  the  plan  evaluation  process.  Ranking  plans  of  action 
in  terms  of  goals  achieved  and  forces  lost  is  the  function  of 
the  plan  evaluation  process  while  the  assignment  of  CAS 
aircraft  to  particular  forward  air  controllers  (FAC)  is 
handled  through  the  sortie  assignment  process  (Ref  6:66). 

Signature  table  techniques  are  applied  to  all  of  the  above 
processes  in  the  general  manner  described  in  Chapter  II. 

An  example  of  this  application  of  the  CAS  target  ranking 
process  area  is  the  set  of  FAC  Target  Importance  Tables. 

FAC  Target  Importance  Tables 

One  purpose  of  a  FAC  is  to  direct  air  strikes  against 
enemy  targets.  When  the  number  of  enemy  targets  exceeds  the 
air  strike  resources  available  for  scheduling  by  the  FAC, 
a  decision  must  be  made  as  to  which  enemy  target  is  most 
important.  The  FAC  might  rank  order  the  targets  and  direct 
strikes  on  the  targets  beginning  at  the  top  of  the  list. 

However,  since  the  battlefield  is  a  dynamic  arena,  the  relative 
importance  of  targets  could  very  well  change  with  time  and 
a  periodic  update  of  the  target  list  would  be  necessary. 

This  process  of  rank  ordering  targets  is  one  of  the  decision 
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processes  modeled  using  signature  table  techniques. 
Target  Characterization 


Each  enemy  target  under  consideration  is  characterized 
in  terms  of  seven  features : 

1.  Proximity  to  friendly  unit  engaged. 

2.  Target  momentum. 

3.  Target  speed. 

4.  Terrain  cover. 

5.  Air  defense  security. 

6.  Aircraft  effectiveness. 

7.  Target  type. 

A  quantization  scheme  is  defined  for  each  of  the  features 
which  maps  the  measured  value,  ,  of  the  feature  into  one 
of  the  quantization  levels  available  for  that  feature. 

The  quantized  values,  are  determined  in  accordance  with 
the  rules  in  Table  I  (Ref  4).  Figure  4  illustrates  the 
structure  of  this  set  of  signature  tables.  The  derived  features 
from  the  three  first  level  tables  (T^ ,  T2,  Tg)  are  used  as 
entries  to  tables  T^  and  Tg.  The  output  of  signature 
table  Tg  is  the  final  evaluation  of  the  target  (target  impor¬ 
tance)  and  is  used  to  rank  order  the  available  targets.  Each 
of  the  signature  tables  T^  through  Tg  are  displayed  in  Tables 
II  through  VI. 

Example 

The  workings  of  the  FAC  target  importance  tables  can 
now  be  demonstrated.  First,  assume  a  FAC  must  evaluate 
two  targets  to  determine  which  target  will  be  attacked  by  a 
particular  CAS  aircraft  available  to  the  FAC.  A  data  set  for 
each  target  is  constructed  and  input  to  the  table  hierarchy 
for  evaluation.  Assume  the  data  sets  are  as  follows. 


Pig  4.  FAC  Target  Importance  Tables  Structure 


TABLE  I 


Quantization  Rules  for  Target  Characterization 


Measured  Value, 

Quantized  Value, 

0  <X  <  3000  (m) 

1 

3000  <  Xx  <  100,000 

2 

K  -m 

0  <  X,  <10  (-6 - ) 

1 

"  4  -  s 

10  <  X2  <  100 

2 

0  <  X,  <1  (m/s) 

1 

1  -  ~ 

1  <  X3  <  1000 

2 

*  no  terrain  cover 

1 

X^  ■  terrain  cover 

2 

Xj  ■  no  air  defense 

1 

Xj.  ■  air  defense 

2 

0  <  x6  <  H  (1) 

1 

H  <  X6  <  61 

2 

61  <  X6  <  100 

3 

X.  -  Tank 

1 

Xy  -  Armored  Personnel 

Carrier  2 

Xy  *  Heavy  Weapon 

3 

Xy  ■  Artillery 

4 
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TABLE  III 


TABLE  IV 

Signature  Table  3  (Target  Vulnerability  Table) 


TABLE  V 


Signature  Table  4  (Likelihood  of  Destruction  Table) 


D 


3 


D 


4 


( 
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A  ■  (4000,90,10,  no  terrain  cover,  no  air  defense 

protection,  SO,  tank) 

B  ■  (1000,5,  0,  no  terrain  cover,  no  air  defense 

protection,  75,  APC) 

The  quantized  data  sets 

A  -  (2,  2,  2,  1,  1,  2,  1) 

B  -  Cl,  1,  1,  2,  1,  3,  2) 

are  then  entered  in  the  first  level  tables.  uses  (F^ , 

F2,  F?)  which  for  target  A  is  (2,  2,  1).  The  Dj^  corresponding 
to  that  signature  is  5.  For  T2,  the  signature  is  (2,1)  for 
target  A  and  D2  »  3.  The  signature  for  T^  is  (2,  1,  1,  2)  and 

Dj  *  3.  The  T^  signature  type  is  (Dj,  D,)  which,  in  the  case 

of  target  A  is  (3,3)  and  •  2.  In  the  final  table  Ts,  the 
entry  signature  is  (2,5)  and  the  target  importance,  F,  is  8. 
Using  the  same  procedure  on  target  B,  F  *  2.  Target  A,  there¬ 
fore,  would  be  the  target  assigned  to  the  available  CAS 
aircraft. 

Error 

This  process  of  using  the  signature  tables  to  evaluate 
the  available  targets  which  results  in  a  rank  order  list  of 
targets  is  an  attempt  to  explicitly  model  the  DM's  decision 
process  in  terms  of  quantifiable  attributes.  It  is,  in 
effect,  a  rough  approximation  of  the  value  that  a  FAC  places 
on  a  target  characterized  by  the  given  attributes. 
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Pig  6.  Momentum  Quantization  Step  Function 


i 
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The  next  subject  is  the  question  of  error.  Speci fical ly , 
it  is  the  question  of  the  error  that  is  generated  by  using  a 
step  function  to  quantize  those  attributes  or  features  which 
are  continuous. 

For  example,  and  F,,  are  represented  by  the  step 
functions  in  Figures  5  and  6.  The  proximity  and  momentum 
are  not  discrete,  but  are  continuous  variables.  As  such, 
what  kind  of  error  is  generated  by  the  attempt  to  represent 
these  features  by  the  given  step  function?  Treatment  of  this 
question  will  be  the  subject  of  the  next  chapter. 
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IV.  Methodology 


The  objective  of  this  chapter  is  to  show  the  development 
of  the  error  measures  to  be  used  in  Chapter  V.  The  error 
tc  be  measured  is  that  which  is  generated  by  quantizing  the 
input  features  for  a  signature  table  evaluation  instead  of 
using  the  continuous  representation  of  the  measured  feature 
and  a  value  function  evaluation.  This  error  will  be  described 
usingthree  measures  developed  in  this  chapter.  Two  of 
these  measures  are  mean  squared  error  (MSE)  and  percent 
error  CPE)*  The  third  will  be  a  comparison  of  sets  of  target 
rank  order  lists  generated  by  the  two  evaluation  schemes. 

The  starting  point  of  the  error  measure  development  is 
the  derivation  of  a  value  function,  U,  from  data  in  a  particular 
signature  table.  The  MSE  under  two  different  sets  of 
assumptions  concerning  U  is  computed  and  then  converted 
into  a  "noise- to- signal"  percent  error.  Finally,  a  simple 
simulation  model  will  be  used  to  generate  the  sets  of  target 
rank  order  lists  for  comparison.  Further  discussion  of  each 
of  these  steps  follows. 

MSE  Determination 

The  military  value  table,  Table  II,  will  be  the  subject 
of  the  error  measurement.  The  signature  type  evaluated  by 
this  table  consists  of  features  F^,  F2,  and  F^.  The  quantization 
rules  that  convert  the  measured  data  X^,  X2,  and  X7  into  the 
quantized  values  are  listed  in  Table  I.  Some  initial 
assumptions  about  these  measured  data  and  their  quantized 
values  need  to  be  made  at  this  point. 


31 


Assume  there  exists  some  function  which  transforms 
into  the  variable  on  the  range  0  <_  H1  <_  2  and,  G2 
which  transforms  X2  into  the  variable  H2  on  the  range  0  <_  H2  £  2. 
Further,  assume  these  transformation  are  such  that  the 
quantization  rules  for  and  X2  can  be  restated  as  follows: 


°  <  H1  <  i,  Fx  =  i  (9) 
1  <  Hj  <  2,  Fx  -  2  (10) 
0  <  H2  <  i,  F2  =  i  (11) 
1  <  H2  <  2,  F2  -  2  (12) 


Next,  one  of  two  assumptions  about  the  table  output,  D^, 
is  necessary.  The  first  possible  assumption  is  that  is  the 
value  function  evaluated  at  the  quantized  values  (F^,  F2,  F7). 
That  is , 

D1CF1,  F2,  F7)  -  UCFX,  F2,  F7)  (13) 

The  other  possible  assumption  is  that  D1  is  the  value  function 
evaluated  at  some  point  (h^  h2,  F?)  that  lies  within  the 
specified  quantization  level: 

DjCFj,  F2,  F?)  -  U(hx,  h2,  F?)  (14) 

Note  here  that  X7  is  a  discrete  variable  and  cannot  be 
represented  by  a  continuous  variable,  hence  it  will  be 
carried  through  the  analysis  with  its  quantized  value  F7. 
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For  example,  consider  a  single  feature  example  where  the 
signature  type  consists  of  a  feature  with  transformed  value 
H.  Then  under  the  first  assumption,  the  table  output,  D, 
at  some  quantized  feature  value  F  would  equal  U(F).  If, 
however,  the  second  assumption  were  true,  D  would  equal  U(h) 
where  h  was  some  value  in  the  domain  of  H  to  which  the 
quantization  rules  assigned  the  value  F  as  shown  in  Figure  7 
One  change  was  made  to  the  original  military  value  table 
before  the  value  function  U  was  derived.  More  specifically, 
a  change  was  made  to  the  quantization  rules  applied  to  X^. 

It  was  discovered  that  by  rearranging  the  values  assigned 
to  Xy,  the  table  output  becomes  monotonically  increasing 
for  two  combinations  of  F^  and  F2  ar*d  minimized  the  positive¬ 
negative  slope  changes  in  the  other  two  combinations.  The 

quantization  rules  were  changed  to  the  system  shown  in 

» 

Table  VII,  where  Fy  and  Fy  are  the  old  and  new  quantized 
values,  respectively.  The  resulting  reordered  military 
value  table  is  illustrated  in  Table  VIII. 

The  data  from  this  new  table  was  used  to  generate  the 
value  function  U  which  was  assumed  to  have  the  general 
form: 

U(x,y,z)  =  Ax  +  By  +  Cz  +  Dxy  +  Exz  +  Fyz  +  G  (15) 

where 

x  -  F, 

y  -  F2 

Z  -  Fy 

The  method  of  least  squares  was  used  for  this  derivation  and 
the  results  are  presented  in  Chapter  V. 


Fig  7.  Single  Feature  Example 


TABLE  VII. 


Revised  Quantization  Rules 


X7 

F7 

f 

F7 

UNK 

5 

1 

HVWP 

3 

2 

ARTY 

4 

3 

MADA 

7 

4 

TANK 

1 

5 

APC 

2 

6 

OADA 

6 

7 

FADA 

8 

8 
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TABLE  VIII 


Revised  Signature  Table  1 
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Once  the  function  U  was  determined,  error  measurement 
could  be  made  under  each  of  the  stated  assumptions  about 
the  table  output.  For  ease  of  reference,  the  case  where 

I  f 

=  U(F1,  F2,  F7  )  will  be  Case  1  and  where  =  Ufh^,  h2,  F?  ) 
will  be  Case  2. 

f 

In  Case  1,  U(F^,  F2,  F^  )  should  equal  the  table  values 
Dj.  However,  since  the  method  of  least  squares,  in  this 
case,  does  not  yield  a  perfect  fit,  it  will  be  assumed 
that  U  is  exact  and  a  new  table  constructed  to  match  U 

f 

evaluated  at  the  quantized  values  of  F^,  F2 ,  and  F7  .  For 

1 

Case  2,  since  does  not  equal  UCF^,  F2,  F^  ),  the  least 
squares  fit  using  the  table  data  represents  an  approximation 
to  U  which  includes  error  from  the  imperfect  fit  and  from 

•  f 

using  Fj^,  F2,  and  F?  values  instead  of  h^,  h2>  and  F7 
as  input  data.  For  the  purposes  of  this  study,  it  will  be 
assumed  under  Case  2  that  the  derived  function  U  is  the  true 
value  function  and  the  table  values  are  the  results  of 

t 

evaluating  U  at  the  point  (h^,  h2,  F7  ). 

In  either  case,  the  parameter  of  interest  is  MSE,  which 
is  defined  as  the  expected  value  of  the  error  squared  where 

f 

error  is  the  difference  between  U(Hj,  H2 ,  F7  )  and 
D1(F1,  F2,  F?’)  (Ref  7:201). 

Expressed  mathematically: 

MSE  -  E[|UCHlf  H2,  F7’)  -  D1(F1,  F2,  F7')|2]  (16) 

t 

If  Hj,  H2,  and  F7  are  independent  random  variables 
with  probability  density  function  as  follows. 
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w  - 


i  ,  0  <  Hj  <  2 


0  ,  elsewhere 


W  - 


J  ,  0  <  H,  <  2 


0  ,  elsewhere 


PC.F7  )  = 


f  *  F7  =  1.2, 3, 4, 5, 6, 7, 8 
0  ,  elsewhere 


then  MSE  can  be  expressed  as 


F?  =1  0  0 


[U(H1,H2,F7  )-dx  (F1  ,F2,F7’  )  fi1  (Hj)  f2CH2)dH1dH2 


F7  ”8  f2,2 


MSF  *  -yj-  I  j  j  CU(H1,H2,F7')-D1(F1,F2,F7')]2dH1dH2  (20) 
F7  81  0  0 

This  expression  must  be  broken  down  as  follows  for 


evaluation 


8  XX 

-j?-  l  ||  [U(H1,H2,F7,)-D1(l,l,F7,)]2dH1dH2 

Fy7®1  J0  0 


+  3T 


8  x  2 

J2"  I  f  [U(H1,H2,F7,)-D1(2,l,F7,)]2dH1dH2 


f7'-i  0  1 
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8  2  1 

*  -JT  I  ff  CUCH1,H2,F7,)-D1(l,2,F7')]2dH1c 

f7'=i  i  o 


8  2  r2 

-£-!]]  [UCH1,H2,F7,)-D1(2,2,F7,)]2dH1dH2  (21) 


F7'=l  0  0 


Substituting  the  assummed  general  form  for  U(H^,H0,F7  ) 
and  performing  the  integration  and  summation  over  the  four 
regions  yields: 

2 

12|-  +  64AB  +  288AC  +  288BC  +  ™AD  +  384AE  +  288AF  +  64AG  + 


128B_  +  116BD  +  288BE  +  384BE  +  384FB  +  64BG  +  816C2  +  288CD  + 


1632CE  +  1632CF  +  288CG  +  1088F2  +  -5-n2—  +  384DE  +  384DF  + 


64DG  +  1088E2  +  1632EF  +  288EG  +  288FG  +  32G2 


l  D1(1,1,F7’)  [A  +  B  +  2CF?  D  +  j  +  EF?  +  FF?  +2G-D1(1,1pF7  )] 
F7’  =  l 

§  *  »  tn  f  »  * 

l  D1(2,l,F7  )  [3A+B+2CF?  +  -^  +  3EFy  +FF?  +2G-D1 (2 , 1 , F?  )] 

F7'  *1 

§  •  »  tn  *  *  ' 

l  D1(1,2,F7  )  [A+3B+2CF7  +  ^  +  EF?  +3FF?  +2G-DX (1 , 2 „ F?  )] 

f7'«i 

§  »  »  on  *  i  i 

l  D1(2,2,Fy  )[3A+3B+2CFy  +  ^  +3EFy  +  3FF?  +  2G-D  (2,2,F7  )  ] 

F’-l 

* 
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The  numerical  results  under  Case  1  and  Case  2  are  different 
and  are  contained  in  Chapter  V. 

Percent  Error 

Since  MSE,  as  applied  to  this  problem,  is  simply 
a  number  with  no  real  meaning  other  than  smaller  is  better, 
some  way  of  attaching  meaning  was  required.  This  was 
accomplished  in  two  ways.  The  first  was  a  percentage  error 
measure  which  was  defined  as 

PE  =  x  100  (23) 

where  MSS  is  the  mean  squared  signal.  The  MSS  under  Case  1 
and  Case  2  assumptions  are  equal  and  can  be  expressed  as 


MSS 


U2CH1,H2,F7’)dH1dH2 


(24) 


After  substituting  the  assummed  general  form  of  U(H^,H2»F7  ), 
MSS  is  as  follows 


MSS  =  -12^-  +  64AB  +  288AC  +  2  -^-AD  +  384AE  +  288AF 

+  6 4 AG  +  —  y-2  +  288BE  +  25|^D  +  288BE 

+  384BF  +  64BG  +  816C2  +  288CD  +  1632CE  +  1632CF 
51  3D2 

+  288CG  +  -■%=-  +  384DE  +  384DF  +  64DG 
+  1088E2  +  1632EF  +  288EG  +  1088F2  +  288FG  +  32G2  (25) 
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PE,  then,  will  give  a  measure  of  the  error  expressed  as  a 
percentage  of  the  assumed  evaluation  function  under  each  case 
of  assumptions. 

Evaluation  Simulation 

The  second  way  of  attaching  meaning  to  the  results  is  to 
generate  some  data  sets  and  evaluate  them  using  each  method 
under  each  assumption  case.  In  order  to  do  that,  however,  a 
second  evaluation  function  had  to  be  determined.  Since 
the  output  of  table  1  is  used  as  an  input  to  signature  table 
5,  a  continuous  output  from  signature  table  1  would  either 
have  to  be  quantized  for  use  by  table  5  or  signature  table  5 
needed  to  be  treated  like  table  1.  The  latter  method  was 
chosen  since  re-quantizing  the  output  from  table  1  to  match 
the  allowable  input  for  signature  table  5  would  destroy  the 
error  information  of  concern.  Signature  table  5  allows  five 
quantization  levels  from  table  1  which  means  the  output  of 

t 

)  would  be  quantized  back  into  five  levels  and 

1 

any  error  increase  (or  decrease)  from  using  F7  ) 

instead  of  the  signature  table  would  get  canceled.  A  general 

1 

form  for  U^(D4,  ))  was  assumed  to  be  Ax+By+C  and 

a  least  squares  fit  obtained.  Using  the  same  procedures  as 
for  signature  table  1,  MSE  and  PE  were  computed  and  are  contained 
in  Chapter  V. 

An  evaluation  program  was  written  to  generate  target  data 
sets,  evaluate  them  using  both  methods  and  assumption  cases, 
and  rank  order  the  results  of  the  evaluation  and  is  contained 
in  Appendix  A.  The  data  sets  were  generated  through  the  use 
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of  a  uniform  random  variate  generator  and  scaled  to  the  appro¬ 
priate  range.  For  the  signature  table  evaluation,  each  data 
set  was  quantized  according  to  the  quantization  rules  in  Table  I. 
For  the  function  evaluation,  the  first  two  features  were 
kept  in  their  continuous  form  while  the  remaining  five  were 
quantized.  The  structure  of  the  function  evaluation  is 
presented  in  Figure  8. 

One  final  measure  was  included  in  the  evaluation  program. 

The  Spearman  rank  correlation  coefficient  was  computed  and 
used  as  a  measure  of  association  between  the  rank  ordering 
using  the  two  methods.  Further  discussion  of  its  use  is 
presented  in  Chapter  V. 

In  summary,  a  theoretical  error  measure,  MSE,  was  developed. 
This  measure  was  then  converted  into  a  percentage  error  measure, 
PE,  which  yields  an  error  measure  which  is  intuitively  more 
meaningful.  To  reinforce  that  meaning,  a  sample  of  30  targets 
is  evaluated  and  rank  ordered  for  comparison  by  each  of  the 
two  evaluation  schemes.  Finally,  a  measure  of  association 
is  used  to  determine  if,  in  fact,  there  exists  a  significant 
difference  in  the  rank  ordering  which  results  from  each  scheme. 
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V.  Results 


In  this  section,  first  the  MSE  for  signature  table  1  and 
5  will  be  presented  under  Case  1  assumptions.  This  will 
be  followed  by  MSE  for  both  tables  under  Case  2  assumptions. 
Next,  the  computation  of  the  percent  error  for  both  cases 
followed  by  the  evaluation  simulation  results.  The  last 
item  that  will  be  covered  is  Spearman's  rank  order  correlation 
coefficient . 

MSE 

Under  the  Case  1  assumptions,  the  table  output  of  a 
given  table  represents  some  value  function  evaluated  at  the 
quantized  values.  The  least  squares  fit  of  data  in  signature 
table  1  to  the  general  form  of  equation  (15)  yielded 
the  following  coefficients . 

A  =  1.6339286 
B  =  -.1160714 
C  =  .7708333 
D  =  .6250000 
E  =  -.1130952 
F  =  -.1964286 
G  =  -2.3437500 

Using  the  resulting  equation  (15)  a  few  signature  table  was 
generated  and  is  presented  in  Table  IX.  The  effect  of  this 
substitution  is  that  U(H^,  H^,  F^')  is  now  an  exact  fit  to 
the  new  military  value  table  and  quantization  error  under 
these  conditions  can  now  be  determined.  The  same  procedure 
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TABLE  IX 


Signature  Table  1  Revised  for  Case  1 


.2604 
.7217 
1.1830 
1.6443 
2.1057 
2.5670 
3.0283 
3.4896 
.  5729 
.  8378 

1.1027 
1.3676 
1.6324 
1.8973 
2.1622 
2.4271 

2.4063 
2.7545 

3.1027 
3.4509 
3.7991 
4.1473 

4.4955 
4.8438 
3.3438 

3.4955 
3.6473 
3.7991 
3.9509 

4.1027 
4.2545 

4.4063 
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was  applied  to  signature  table  5  for  the  reasons  given  in 
Chapter  IV.  The  assumed  form  of  the  value  function  was 
Ax  +  By  +  C  where  X  =  and  y  =  u(Hj,  ).  The 

coefficients  generated  by  a  least  squares  fit  were 

A  =  1.9 
B  =  1.4333 
C  =  -3.1 

The  revised  signature  table  5  is  presented  in  Table  X. 
Equation  (14),  using  the  adjusted  signature  table  1  values 
reduces  to 

MSE  =  (186.6841  -  148.4483) 

MSE  =  1.1949 

The  MSE  for  signature  table  5  is  computed  in  the  same  way 
and  yields 

MSE  =  -y-  (375.4919  -  365.0954) 

MSE  =  0.6931 

For  Case  2  assumptions,  the  derived  value  function  is 
again  assumed  to  be  correct  and  the  signature  table  values 
are  the  result  of  evaluating  the  function  at  points 
other  than  the  quantized  feature  values.  The  mean  squared 
error  under  Case  2  for  signature  table  1  is 

MSE  =  —  (186.6841  -  134.7680) 

32 

MSE  =  1.6224 
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TABLE  X 


Signature  Table  5  Revised  for  Case  1 


15 

°4 

15 

D1 

15 

F 

1 

1 

.2333 

1 

2 

1.6666 

1 

3 

3.0999 

1 

4 

4.5332 

1 

5 

5.9665 

2 

1 

2.1333 

2 

2 

3.5666 

2 

3 

4.9999 

2 

4 

6.4332 

2 

5 

7.8665 

3 

1 

4.0333 

3 

2 

5.4666 

3 

3 

6.8999 

3 

4 

8.3332 

3 


5 


9.7665 


Once  again,  the  same  computation  for  the  original  signature 
table  5  yields 

MSE  =  (375. 4919-360. 9513) 

MSE  =  .9694 

It  can  be  seen  that  under  Case  1  assumptions,  where  the 
table  output  was,  in  fact,  the  value  function  evaluated  at 
the  quantized  features ,  the  MSE  was  better  (smaller)  than 
when  the  table  output  was  determined  as  in  Case  2. 

Percent  Error 

Recall  from  Chapter  IV,  equation  (23)  which  defined 
a  percent  error  measure.  This  measure  expresses  the  quan¬ 
tization  error,  MSE,  as  a  percent  of  the  mean  squared 
signal,  MSS  (equation  24).  Under  Case  1,  MSS  and  PE  for 
signature  table  1  are 


MSS  * 

(186.6841) 

MSS  =* 

5.8339 

PE  - 

1.1949 

$.8339  x  100 

PE  - 

20.48% 

The  MSS  and  PE  for  signature  table  5  are 
MSS  »  25.0328 
PE  -  2.771 
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Applying  Case  2  assumptions,  the  results  changed  as 
follows.  For  signature  table  1 

MSS  =  5.8339 

PE  =  27.815 

Calculations  for  signature  table  5  yield 
MSS  =  25.0328 
PE  =  3.875 

These  results  indicate  that  in  either  assumption 
case,  the  initial  feature  quantization  error  is  at  least  205  (PE 
for  table  1  under  Case  1).  Signature  table  5  only  adds 
another  3-45  to  the  initial  quantization  error.  The 
evaluation  simulation  was  written  in  an  attempt  to  give 
a  better  feel  for  what  the  results  thus  far  mean. 

Evaluation  Simulation 

Under  Case  1  assumptions,  the  evaluations  and  rank  order 
listings  of  the  30  sample  targets  are  presented  in  Table  XI 
for  both  the  signature  table  and  value  function  methods. 

Table  XII  contains  the  same  information  for  Case  2  assumptions. 

A  significant  result  in  the  signature  table  ranking  is  the 
number  of  ties  that  resulted  under  Case  2  assumptions. 

Out  of  30  targets  evaluated,  a  total  of  28  involved  ties. 

This  indicates  that  this  method  of  evaluation  must  include 
a  means  of  settling  ties.  A  tie  between  the  evaluation  of 
two  or  more  targets  defeats  the  purpose  of  the  evaluation 
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TABLE  XI 

Case  1  Rank  Order  List 


Target  Signature  Table  Value  Function 


Number 

F 

RANK 

u 

RANK 

1 

5.9 

24.0 

2.8620 

14.0 

2 

1.5 

5.0 

-.0845 

6.0 

3 

4.2 

17.0 

4.3709 

24.0 

4 

.5 

1.0 

-.6225 

5.0 

5 

6.1 

26.0 

4.4993 

26.0 

6 

2.7 

7.0 

2.0878 

10.0 

7 

5.6 

22.5 

3.7377 

22.0 

8 

3.4 

12.0 

3.5396 

20.0 

9 

5.1 

19.0 

4.5613 

27.0 

10 

6.  8 

29.0 

5.4545 

29.0 

11 

6.1 

26.0 

4.1362 

23.0 

12 

3.0 

9.0 

2.9885 

16.0 

13 

1.1 

2.5 

-.7897 

3.0 

14 

4.2 

18.0 

3. 3920 

19.0 

15 

5.5 

21.0 

2.7129 

12.0 

16 

3.8 

14.0 

2.7735 

13.0 

17 

3.0 

9.0 

1.9911 

9.0 

18 

6.1 

26.0 

3.3367 

18.0 

19 

8.7 

30.0 

7.4672 

30.0 

20 

4.0 

15.0 

.  5291 

7.0 

21 

6.4 

28.0 

5.0821 

28.0 

22 

4.1 

16.0 

2.9863 

15.0 

23 

1.5 

5.0 

-1.6174 

1.0 

24 

1.1 

2.  5 

-.7503 

4.0 

25 

5.6 

22.5 

3.2957 

17.0 

26 

5.2 

20.0 

4.4379 

25.0 

27 

3.8 

13.0 

3.5408 

21.0 

28 

3.0 

9.0 

1.7593 

8.0 

29 

1.5 

5.0 

-1.3775 

2.0 

30 

3.1 

11.0 

2.6527 

11.0 
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TABLE  XII 


Case  2  Rank  Order  List 


Target 

Number 


Signature  Table 


F 

RANK 

U 

7.0 

23.5 

2.8620 

2.0 

8.0 

- .0845 

7.0 

23.5 

4.3709 

1.0 

2.0 

-.6225 

8.0 

27.5 

4.4993 

2.0 

8.0 

2.0878 

5.0 

19.5 

3.7377 

2.0 

8.0 

3.5396 

7.0 

23.5 

4.5613 

7.0 

23.5 

5.4545 

8.0 

27.5 

4.1362 

2.0 

8.0 

2.9885 

1.0 

2.0 

-.7897 

4.0 

15.5 

3.3920 

5.0 

19.5 

2.7129 

3.0 

13.0 

2.7735 

2.0 

8.0 

1.9911 

8.0 

27. S 

3.3367 

9.0 

30.0 

7.4672 

4.0 

15.5 

.5291 

8.0 

27.5 

5.0821 

4.0 

15.5 

2.9863 

2.0 

8.0 

-1.6174 

1.0 

2.0 

-.7503 

5.0 

19.5 

3.2957 

5.0 

19.5 

4.4379 

4.0 

15.5 

3.5409 

2.0 

8.0 

1.7593 

2.0 

8.0 

-1.3775 

2.0 

8.0 

2.6527 

Value  Function 


RANK 


14.0 

6.0 

24.0 

5.0 

26.0 

10.0 

22.0 

20.0 

27.0 

29.0 

23.0 

16.0 

3.0 

19.0 

12.0 

13.0 

9.0 

18.0 

30.0 

7.0 

28.0 

15.0 

1.0 

4.0 

17.0 

25.0 

21.0 

8.0 

2.0 

11.0 
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if  the  only  targets  to  be  ranked  are  tied.  At  best,  the 
effectiveness  of  the  system  is  drastically  impaired  assumming 
not  all  the  targets  to  be  ranked  tie  in  their  evaluations. 

The  purpose  of  the  evaluation  is  to  prioritize  the  allocation 
of  weapon  resources  and  if  the  evaluation  method  used  yields 
a  tie,  the  decision  maker  is  left  at  square  one. 

This  apparent  lack  of  fidelity  in  the  evaluation  from 
the  signature  table  method  under  Case  2  assumptions  is  not 
a  problem  in  the  value  function  method  and  is  much  less 
significant  in  the  signature  table  method  under  Case  1 
assumptions.  The  value  function  method  yielded  no  ties 
while  the  Case  1  signature  table  method  yielded  only  13  of 
the  30  targets  involved  in  ties.  In  both  cases,  the 
fidelity  of  the  evaluation  systems  is  improved  over  the 
Case  2  signature  table  method. 

The  data  sets  that  were  evaluated  in  the  previous  runs 
were  generated  by  assuming  each  of  the  seven  features  were 
uniformly  distributed  across  the  appropriate  ranges.  In 
a  further  attempt  to  visualize  the  results  of  quantizing 
the  input  variables,  two  more  sets  of  data  were  generated. 

In  one  set,  features  and  X2  were  drawn  from  an  exponential 
distribution  with  mean  =  1,  while  the  other  features  were 
generated  as  before.  The  computer  program  modification 
is  contained  in  Appendix  B.  The  results  are  contained  in 
Tables  XIII  and  XIV  for  Case  1  and  Case  2  assumptions , res¬ 
pectively.  The  signature  table  method  under  Case  1 
resulted  in  16  of  30  targets  involved  in  ties.  Under  Case  2, 
24  of  the  30  targets  were  involved  in  ties. 


TABLE  XIII 


Case  1  Rank  Order  List 

(Exponential) 

Target 

Number 

Signature 

F 

Table 

RANK 

Value 

\J 

Function 

RANK 

1 

0.5 

2.5 

-.2609 

5.0 

2 

4.1 

21.0 

1.2579 

9.0 

3 

7.6 

29.0 

6.9179 

28.0 

4 

4.0 

19.0 

2.3891 

14.0 

5 

3.1 

12.5 

2.1499 

13.0 

6 

3.1 

12.5 

3.2631 

20.0 

7 

3.1 

12.5 

1.8421 

10.0 

8 

4.4 

24.0 

4.8327 

26.0 

9 

2.3 

8.0 

-.1612 

7.0 

10 

3.1 

16.0 

2.9027 

17.0 

11 

3.1 

12.5 

2.0188 

11.0 

12 

3.7 

17.5 

3.9516 

23.0 

13 

6.1 

26.0 

5.9806 

27.0 

14 

1.1 

5.0 

-1.4396 

3.0 

15 

1.1 

4.0 

-1.0317 

4.0 

16 

-.2 

1.0 

-2.2308 

1.0 

17 

6.1 

26.0 

4.6451 

24.0 

18 

3.1 

12.5 

2.4397 

15.0 

19 

6.9 

28.0 

7.0586 

29.0 

20 

.5 

2.5 

-.2458 

6.0 

21 

3.7 

17.5 

3.0716 

18.0 

22 

1.5 

6.0 

-1.5888 

2.0 

23 

4.1 

21.0 

3.2105 

19.0 

24 

4.2 

23.0 

3.5385 

21.0 

25 

2.7 

9.0 

2.0788 

12.0 

26 

1.9 

7.0 

.7075 

8.0 

27 

9.0 

30.0 

8.9918 

30.0 

28 

6.1 

26.0 

4.6915 

25.0 

29 

4.1 

21.0 

3.6414 

22.0 

30 

3.1 

12.5 

2.8702 

16.0 

TABLE  XIV 

Case  2  Rank  Order  List 


Target 

NUMBER 

Signature 

F 

Table 

RANK 

1 

1.0 

2.5 

2 

4.0 

20.5 

3 

7.0 

27.0 

4 

4.0 

20.5 

5 

2.0 

10.0 

6 

2.0 

10.0 

7 

2.0 

10.0 

8 

4.0 

20.5 

9 

2.0 

10.0 

10 

3.0 

16.0 

11 

2.0 

10.0 

12 

4.0 

20.5 

13 

7.0 

27.0 

14 

1.0 

2.5 

15 

2.0 

10.0 

16 

1.0 

2.5 

17 

7.0 

27.0 

18 

2.0 

10.0 

19 

7.0 

27.0 

20 

1.0 

2.5 

21 

4.0 

20.5 

22 

2.0 

10.0 

23 

4.0 

20.5 

24 

4.0 

20.5 

25 

2.0 

10. C 

26 

2.0 

10.0 

27 

10.0 

30. C 

28 

7.0 

27.  C 

29 

4.0 

20. £ 

30 

2.0 

10.  C 

(Exponential) 


Value  Function 
U  RANK 


-.2609 

5.0 

1.2579 

9.0 

6.9179 

28.0 

2.3891 

14.0 

2.1499 

13.0 

3.2631 

20.0 

1.8421 

10.0 

4. 8327 

26.0 

-.1612 

7.0 

2.9027 

17.0 

2.0188 

11.0 

3.9516 

23.0 

5.9806 

27.0 

-1.4396 

3.0 

-1.0317 

4.0 

-2.2308 

1.0 

4.6451 

24.0 

2.4397 

15.0 

7.0586 

29.0 

-.2458 

6.0 

3.0716 

18.0 

-1.5888 

2.0 

3.2105 

19.0 

3.5385 

21.0 

2.0788 

12.0 

.7075 

8.0 

8.9918 

30.0 

4.6915 

25.0 

3.6414 

22.0 

2.8702 

16.0 

The  second  additional  set  of  data  was  generated  by  taking 
X^  and  from  a  normal  distribution  with  mean  of  0.75  and 
standard  deviation  0.25.  The  required  program  modification 
is  contained  in  Appendix  C.  The  results  of  these  runs  are 
contained  in  Tables  XV  and  XVI.  Using  the  signature  table 
method  under  Case  1  assumptions,  17  of  the  30  targets  evaluated 
were  tied.  Under  Case  2  assumptions,  29  of  the  30  targets 
resulted  in  ties.  The  value  function  method  in  both  additional 
data  sets,  of  course,  yielded  no  ties. 

Once  again,  the  lack  of  fidelity  in  the  signature  table 
method  under  either  set  of  assumptions  becomes  apparent. 

In  all  three  data  sets,  the  signature  table  method  under 
Case  2  assumptions  results  in  a  significant  number  of  ties. 
These  ties,  as  stated  before,  provide  no  real  solution  to 
the  DM's  rank  ordering  problem. 

Spearman  Rank  Correlation  Coefficient 

The  last  measure  of  merit  to  be  applied  to  these  methods 
of  target  evaluation  and  ranking  comes  from  the  field  of 
non-parametric  statistics.  This  measure,  Spearman's 
rank  correlation  coefficient,  can  be  used  to  test  the 
degree  of  association  between  series  of  rank  ordered  data. 

The  rank  correlation  coefficient,  rg ,  is  computed  using  the 
following  equation 


n 

61  d. 


2 


rs  -  1  -  i=l  C26) 

n(n2-l) 

where  d^  is  the  difference  in  the  ranks  of  the  same  target 
by  the  two  methods  being  compared  (Ref  2:550).  The  rank 
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TABLE  XV 


Case  1  Rank  Order  List  (Normal) 


Target 

Number 

Signature 

F 

Table 

RANK 

Value  Function 

U  RANK 

1 

1.1 

2.5 

-1.0041 

2.0 

2 

2.7 

17.0 

1.3397 

11.0 

3 

4.4 

21.5 

4.4936 

25.0 

4 

2.5 

16.0 

2.2792 

18.0 

5 

4.4 

21.5 

4.3279 

24.0 

6 

5.0 

25.0 

5.2359 

26.0 

7 

5.6 

26.0 

3.6160 

21.0 

8 

1.8 

10.5 

1.7026 

13.0 

9 

4.4 

21.5 

3.2646 

19.0 

10 

2.4 

13.5 

1. 7290 

15.9 

11 

8.9 

30.0 

8.0302 

30.0 

12 

2.4 

13.5 

1.7036 

14.0 

13 

2.4 

13.5 

2.1417 

17.0 

14 

5.7 

27.5 

5.9257 

27.0 

15 

1.2 

4.0 

-.0985 

5.0 

16 

1.1 

2.5 

-1.2198 

1.0 

17 

.5 

1.0 

.0579 

6.0 

18 

2.4 

13.5 

1.1676 

10.0 

19 

1.5 

5.0 

.9480 

9.0 

20 

1.7 

8.0 

-.1529 

4.0 

21 

6.9 

29.0 

6.0758 

28.0 

22 

1.7 

8.0 

-.3551 

3.0 

23 

1.8 

10.5 

.5340 

7.0 

24 

4.4 

21.5 

3.6504 

22.0 

25 

5.7 

27.5 

6.4688 

29.0 

26 

3.7 

19.0 

3.3972 

20.0 

27 

1.7 

8.0 

.6912 

8.0 

28 

1.6 

6.0 

1. 3466 

12.0 

29 

3.4 

18.0 

4.1435 

23.0 

30 

4.6 

24.0 

1.8486 

16.0 
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TABLE  XVI 


Case  2  Rank  Order  List  (Normal) 


Target 

Number 

Signature 

F 

Table 

RANK 

Value  Function 

J  RANK 

1 

2.0 

11.0 

-1.0041 

2.0 

2 

2.0 

11.0 

1.3397 

11.0 

3 

4.0 

21.5 

4.4936 

25.0 

4 

2.0 

11.0 

2.2792 

18.0 

5 

4.0 

21.5 

4.3279 

24.0 

6 

5.0 

25.5 

5.2359 

26.0 

7 

5.0 

25.5 

3.6160 

21.0 

8 

2.0 

11.0 

1.7026 

13.0 

9 

4.0 

21.5 

3.2646 

19.0 

10 

2.0 

11.0 

1.7290 

15.0 

11 

9.0 

30.0 

8.0302 

30.0 

12 

2.0 

11.0 

1.7036 

14.0 

13 

2.0 

11.0 

2.1417 

17.0 

14 

7.0 

28.0 

5.9257 

27.0 

15 

1.0 

2.0 

-.0985 

5.0 

16 

2.0 

11.0 

-1.2198 

1.0 

17 

1.0 

2.0 

.0579 

6.0 

18 

2.0 

11.0 

1.1676 

10.0 

19 

2.0 

11.0 

.9480 

9.0 

20 

2.0 

11.0 

-.1529 

4.0 

21 

7.0 

28.0 

6.0758 

28.0 

22 

2.0 

11.0 

-.3551 

3.0 

23 

2.0 

11.0 

.5340 

7.0 

24 

4.0 

21.5 

3.6504 

22.0 

25 

7.0 

28.0 

6.4688 

29.0 

26 

4.0 

21.5 

3.3972 

20.0 

27 

2.0 

11.0 

.6912 

8.0 

28 

1.0 

2.0 

1.3456 

12.0 

29 

2.0 

11.0 

4.1435 

23.0 

30 

4.0 

21.5 

1.8486 

16.0 

56 


correlation  coefficient  can  be  used  in  this  situation  to 
determine  if  there  exists  a  statistical  difference  between 
the  evaluation  methods  under  consideration.  Specifically, 
the  null  hypothesis  was  no  association  between  the  two 
methods.  Acceptance  of  the  null  hypothesis  would  imply  a 
significant  difference  existed  in  the  valuation  methods. 

The  critical  values  of  rg  for  a  sample  size  of  30  are  0.305 
and  0.432  for  a  0.95  and  0.99  confidence  level  (Ref  2:A46). 

The  null  hypothesis  of  no  association  would  then  be  rejected 
if  rg  was  greater  than  or  equal  to  the  appropriate  critical 
value.  Under  Case  1  assumptions,  calculations  yield  rg  =.8396 
which  is  well  beyond  the  rejection  region  for  both  confidence 
levels.  The  value  of  rg  for  Case  2  assumptions  is  .8254,  also 
well  into  the  rejection  region.  The  null  hypothesis  of 
no  association  would  be  rejected  under  either  set  of 
assumptions  at  the  0.01  level  of  significance.  This  result 
means  that  the  signature  table  method  of  evaluation  does 
result  in  a  rank  order  listing  of  targets  that  is  associated 
to  a  statistically  significant  level  with  the  continuous 
value  function  evaluation.  A  measure  of  that  degree  of 
association  has  already  been  presented  as  the  MSE  and 
subsequent  expression  as  percent  error. 

The  application  of  this  same  measure  to  the  additional 
data  sets  yields  similar  results.  These  results  are 
summarized  in  Table  XVII. 
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TABLE  XVII 


Spearman  Rank 

Correlation 

Coefficients 

Data  Set 

Case  1 

Case  2 

Uniform 

0.8396 

0.9254 

Exponential 

0.9211 

0.8776 

Normal 

0.9218 

0.8249 

In  all  cases,  the  null  hypothesis  can  be  rejected  and,  there¬ 
fore, the  signature  table  representation  of  the  value  function 
results  in  evaluations  that  are  statistically  similar. 
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VI.  Conclusions  and  Recommendations 


The  signature  table  is  a  representation  of  a  value 
function  which  has  been  evaluated  at  certain  input  levels. 

This  is  the  major  assumption  of  this  study  and  if  it  is 
valid,  significant  benefits  could  be  derived  form  replacing 
the  signature  table  with  the  value  function  it  represents. 

First,  the  fidelity  of  the  resultant  figure  would  be  greatly 

improved.  For  instance,  using  signature  tables  in  the  evaluation 

of  targets  as  presented,  yeilded  a  much  higher  probability  of 

a  tie  between  targets  than  did  use  of  the  assumed  value 

function.  Since  the  purpose  of  the  scheme  is  to  simulate  a 

FAC  rank  ordering  available  targets  for  strikes,  a  ties  does 

no  good.  The  decision  maker  will  not  accept  a  tie  in  the 

evaluation  of  targets  and  if  one  results  from  an  artificial 

evaluation  method,  a  tie  breaker  must  be  included.  The 

value  function  method  needs  no  such  tie  breaking  scheme. 

Second,  there  would  be  no  need  for  quantization  rules  to  be 

applied  to  the  measured  continuous  features,  X  ,  in  the  value 

n 

function  evaluation  scheme.  For  those  features  that  are 
discrete  to  begin  with,  such  as  target  type,  a  means  of  quantizing 
the  measured  value  into  data  would  still  be  necessary.  One  of 
the  stated  limitations  of  signature  table  performance  is  that 
it  Mis  directly  related  to  the  discrete  ranges  of  the  input 
features"  (Ref  6:64).  This  limitation  could  be  negated  through 
use  of  a  value  function.  Third,  there  is  the  question  of  storage 
space  required  for  each  of  the  two  methods.  Both  require 
an  input  data  set  array.  However,  the  value  function  method 
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requires  only  storage  of  the  equation  coefficients  and  a  function 
statement  to  perform  the  evaluation  while  the  signature  table 
method  requires  storage  of  the  output  features  and  a  table 
search  routine  to  find  the  required  output.  In  the  case  of 
the  evaluation  simulation  written  for  this  study,  the  signature 
table  method  required  storage  of  96  table  values  and  search 
routine  of  five  lines.  The  value  function  method  required 
storage  of  ten  values  (coefficients)  and  two  function  state¬ 
ments.  The  savings  results  from  the  number  of  values  that 
must  be  stored.  In  this  instance  a  storage  savings  of  almost 
90%  resulted.  This  research  has  shown  that  there  is  sufficient 
merit  to  the  application  of  a  value  function  as  a  replacement 
for  signature  table  evaluation  to  warrant  further  formal 
consideration.  It  has  been  shown,  subject  to  the  stated 
assumptions,  that  there  is  a  quantization  error  that  can  be 
eliminated  by  replacing  the  signature  evaluation  scheme  with 
a  value  function  scheme.  The  benefits  interms  of  improved 
evaluation  fidelity  and  decreased  storage  requirements  have 
been  demonstrated.  The  value  function  method  of  evaluation 
clearly  has  merit  and  should  be  pursued. 

Subject  to  the  stated  assumptions,  these  benefits  and  the 
percentage  error  figures  previously  reported  lead  to  the 
conclusion  that  a  value  function  replacement  for  signature 
table  evaluation  should  be  investigated  in  greater  detail. 
Specifically,  it  is  the  recommendation  of  this  study  that  a 
formal  application  of  multiattribute  utility  assessment 
procedures  such  as  those  described  by  Keeney  and 
Raiffa  to  a  specific  signature  table  evaluation  scheme  be 
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accomplished  (Ref  1).  This  line  of  research  could  have  significant 
impact  on  the  artificial  intelligence  techniques  currently  in 
use  to  model  the  decision  maker  in  computer  simulations. 

The  concept  of  a  decision  maker  being  risk  averse,  neutral,  or 
prone  could  be  incorporated  into  the  simulation  using  utility 
functions  in  place  of  signature  tables  and  the  results  analyzed. 
This  is  not  to  imply  that  treatment  of  risk  is  absent  in  the 
signature  table  method;  however,  it  would  be  more  easily  seen 
and  interpreted  using  utility  functions. 


61 


Bibliography 


1.  Keeney,  Ralph  L.  and  Howard  Raiffa.  Decisions  With 
Multiple  Objectives:  Preferences  and  Value  Tradeoffs. 

New  York:  John  Wiley  and  Sons ,  1976 . 

2.  Mendenhall,  William  and  Richard  L.  Scheaffer.  Mathe¬ 
matical  Statistics  with  Applications.  North  Scituate: 
buxbury  Press,  1973. 

3.  Page,  Carl  V.  "Heuristics  for  Signature  Table  Analysis 

as  a  Pattern  Recognition  Technique,"  IEEE  Transactions 
on  Systems,  Man,  and  Cybernetics,  SMC- 7 (2) :  77- &6 

(February  1977). 

4.  Reiss,  Royce  H. ,  Tactical  Systems  Division,  ACS/Studies 
and  Analysis  (personal  correspondence).  Pentagon, 
Washington  D.C.,  29  August,  1979. 

5.  Samuel,  A.  L.  "Some  Studies  in  Machine  Learning  Using 

the  Game  of  Checkers.  II -Recent  Progress,"  IBM  Journal 
of  Research  and  Development.  6_:  601-617  (November  1967). 

6.  "TAC  ASSESSOR  Methodology  Manual,"  prepared  for  Tactical 
Systems  Division,  ACS/SA,  Headquarters  USAF,  by  General 
Research  Corporation,  December  1978. 

7.  Taub,  Herbert  and  Donald  L.  Schilling.  Principles  of 
Communication  Systems.  New  York:  McGraw-Hill  Book  Co., 

mr. -  - 


62 


Appendix  A 


PROGRA  m  TG'rIMP(INPUT,OUlPUT  ,  TA°f6 ) 

INTEGER  TaPLEl,TA«LE5 
INTEGER  T5PlE2,T8PlE3,TAELE4 
INTEGER  X,XD,FD 

DIMENSION  *C(7),  XO (7) ,  T  A  BL  El <  2 , 2,  3)  ,  T  -*Pl  E2  (2, 8  )  , 

CT  A9LE3 (  * , ’» 3) *  TAPlE4(3,3>,  T  A9LE5  <  3,  H  ,  X<7), 

GRD1 (30) , R3?( 30) ,RC ( 30) ,1k ( 30) ,R(31) ,?*NK *C  ( 30 ) , RANK&9 1 ( 3T ) . 
CRANKR02(30) ,  XOK7  ) 

COMMON  TflnLCi,TASLr2,TA9LE3,TA')LE4,TAPLE' 

801  FORMAT  (813) 

802  FORMAT  (3T7) 

803  FORMAT  (5T3) 

901  FORMAT  (1H1,13X,"  T ARGE  T",?X»"SlGNATURe  TABLE", 9X, 

C" VALUE  FUNCTION",/) 

902  FORMAT  (13X,"  NUMBER",  7  X ,  "F",  1 IX  ,  "PAN  «■",  7  X  ,"U“  ,  1 1<  , 
CRANK",  /,  /) 

9P3  FORMAT  (1BX,  I2,8X,F4.1,9X,F*,.l,«5X,FA.f  >,ru,i) 

904  FORMAT  (1H1) 

905  FORMAT  (  9  (  T3 ,5X)  ,  I  3 ,2  (  5X,  F  8 .4)  , 5  ( 5X  ,T  3)  ) 

READ  (8,801)  < ( <T APLE1 (I , J  ,K)  ,K  =  1,8 ) ,  J=l , 2>  , 1=1, 2) 

READ  (6,60  1)  ((TAPLE2(I»J)  ,  J=  1 , 8)  ,1=1  ,?) 

READ  (9,807)  ((  (TAPLE3(I,J,K)  ,K=1,3),J=1,2)  ,1=1,4) 

READ  (8,3^2)  ( ( TA  PL  E4  ( I ,  J  )  ,  J=  1,  3)  ,  I  =1  ,3) 

READ  (3,30  3)  ((TA?LE5(I,J)  ,J=1,E)  ,1=1  ,3) 

WRITE  9r4 
DO  1 ^ O  N  =  1 , 3  C 

CALL  DATASET  (XC,XD) 

WRITF  9  f»5  ,N,  (X  O  (  j)  ,  j=i  ,  7)  ,  N,  XC  (1  >  ,  >C  (2  ) ,  ( XQ(  J )  «  J  =  ! ,  *) 

CALL  T  A  PD  (XO, 104, pO) 

CALL  r  A  PC  ( XC , I D4, FC) 

RDl(N)=FO 
RC  (N)=FC 
OO  15  1=1,7 
XD1(I)=XD(I) 

15  CONTINUE 

CALL  T A  PC  ( XOl  ,  I D4  ,  FC) 

R02(N)=FC 
100  CONTINUE 

CALL  NMPANK  <RC,V,:i.C,IR,R,RANKRC,S,T> 

CALL  NMRANK  (ROl , N, 0. 0 , IR , R,R ANKROl ,S  ,T) 

CALL  NMRANK  (RD2,  N,  0. 0  ,  IP  ,  R,RANKP.D2 , 9  ,T ) 

WRITE  901 
WRITE  902 
00  300  1=1, S 

WRITE  90  3,1,  RD2  (I )  ,  RANKRD  2(1)  ,RC(I)  ,R  ANK  "C  f  I) 

300  CONTINUF 
WRITE  901 
WRITE  902 
DO  400  1=1, N 

WRITE  903,I,P.D1(I),  RANKRD  1(1)  ,RC(I)  ,R  ANK  'C(I) 

400  CONTINUE 
D1*D2=C 
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00  40  1=1,  N 

01=01*  (RANKSCC)  -RAN<R OKI))**  2 
02=02*  (  RA  NKRC  (  I )  -RANKP  02(1 ) >••? 
hP  CuNTINUr 

R  1=1-6*01/ (NMN**  2-1)  ) 

R 2= 1-6* 02/ <N* (N*»  2-1) ) 

PRINT*  ,  lu  1 » “Rl=  ”,  Rl»  ”  *?a  "t3? 

203  STOP  “END  OF  TGTT  HP” 

END 

SUBROUTINE  OATASET  (XC,XD) 

DIMENSION  XC  (7)  ,X0<  7) 

INTEGER  XO 
DO  10  1=1,7 
A  =  P.A  Nr  (  0) 

IF  (I.GT.5)  GO  TO  11 

X0<r)=2*A*l 

GO  TO  1 ? 

11  IF  (T.GT.6)  GO  TO  12 

XD(I)=3*A+1 

GO  TO  17 

12  XO (I )  =  8*  A  *1 

13  IF  (I.GT.2)  GO  TO  14 

XC(I)=2*A 

GO  TO  1" 

14  X0(I)=XO(I) 

10  CONTINUE 

RETURN 

ENO 

SUBROUTINE  TABO  (X,I04,FD) 

COMMON  TABLE1,TABLE2,TABLE3,  TABLE*,  Tf.  PLE' 

INTEGER  MBLE1, TABLES 
INTEGER  TABLE2,TA  BLE3,TARLE4 
INTEGER  X ,  FO 

DIMENSION  X  C  7 1  , TAEL  El  (2, 2,0)  ,TA9LE2  (2  ,*)  ,T  A  BL  E3(  4  »  2,  * )  , 
CTA3LE4<3,3) , TABLES (3,5) 

1 01 =TA8LE1 (XC1),X (2),X(7) ) 

I 02=T  AB  LE2 (X (3) ,X (7)) 

IF  (<X<1>  .  £9.1)  .AN0.<X<4>  .EQ.l))  50  TO  S'’ 

GO  TO  31 

30  1=1 

GO  TO  39 

31  IF  <(X(1>  .EO.l)  .AN0.(X(4>  .EQ.2))  GO  TO  3! 

GO  TO  33 

32  1=2 

GO  TO  39 

33  IF  <<X(1>  .EQ.Z)  .AN0.(XC4)  .EQ.2))  GO  TO  3' 

1=3 

GO  TO  39 

34  1*4 

39  1 03*T  ABLE  3  (I,X(S),X(6)) 

I04*TA8LEMI03,I02> 

F0=TA8LF5  CID4,I01) 

RETURN 
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END 

SUBROUTINE  TA8C  <X,ID4,FC> 

0 IM€NSI ON  X(7) 

A=1.6339?Pf 
3=-.U50M«*3 
C=. 7708333 
0  =. 625 

£=-.11309524 
F=-. 196^2857 
S=-2.  34375 

0A=A»X(1)+R*X(2)+C*X(7) 

09=0*X(1)*X(2)+E*X(1)»X(7)  *F*  X(2)*X  (7  ) 

0l=0A*D9+G 

A=1.9 

B=1.h333 

C=-3.1 

FC=A*IO««  +  B¥01+C 
RETURN 
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Appendix  B 


GHRROUTTNt-  QAT4SrT  <YC,X'> 

dimension  xc(7),xn(7) 
INTEGER  VO 
03  10  1=1,’ 

ArPftMC  (  p) 

0= -a  l_  n  n  (A ) 

IF  (I .r,-.2)  GO  TO  11 

yn(i)=f,*i 

IF  <X0 (T)  .GT.2)  GQ  to  lfl 


GO  T n  1  *» 

18 

XO ( I )  =  ? 

GO  TO  1 ’ 

11 

I c  (I.Gt.5) 

xo(i)  =  ?»a+i 

GO  T o  1? 

GO 

T0 

1  3 

13 

Tf  (T.GT.fe) 
XOtlJs’-fl ♦! 
GO  T0  1  ’ 

GO 

TO 

14 

14 

XO(T)=B*A+l 

12 

IF  (T.GT.2) 
XC ( I »  =  ^ 

GO 

T0 

15 

IF  *Y" (I) ,GT 
GO  TO  11 

.?) 

GC 

TO 

16 

xr  (i)  =  ? 

GO  TO  1" 

15 

XC(I)  =  XO(I) 

10 

CONTINtIF 

RETURN 


END 
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Appendix  C 


SUBROUTINE  1ATJ.S-1  tvr,x  ) 
DIMENSION  VS(7),yr'(7) 
INTEGER  XD 
DO  13  1  =  1, x 
A  =  P.ANr  ( D) 

CALL  MORTAL  (.7*, .25, XI) 
IF  (I.GT.2)  GO  TO  11 
XO<I) sxi+l 

IF  (XDf  T)  .GT.2)  GO  TO  16 
IF  ( <0 ( T )  »LT  .0  )  GO  TO  19 
GO  TO  12 

18  XD(I>=? 

GO  tt  i? 

19  xnm=n 
GO  TO  1? 

11  IF  (I.GT.F)  GO  TO  1? 

XD(I) =  2*A*i 

GO  T**  12 

13  IF  (T.G7.6)  GO  TO  14 

XO(I)  =T*A+1 

GO  TO  1  ?. 

14  X1(I)=**A+j 

12  IF  (I.GT.2)  GO  TO  15 

XC(I)  -XI 

IF  (XO (I) ,GT.2 )  GO  TO  It 
IF  (XG(T)  .LT.1 )  GO  TO  17 
GO  TO  ln 

16  XC(I)=? 

GO  TO  10 

17  XC ( T) =  0 
GO  TO  in 

15  xc(i>=xn<n 

10  CONTI  NUr 
RETURN 
ENO 


SUBROUTINE  NORMAL  (EX, STD, XI) 
10  C=RANF(0) 

Vl=2*C-t 

OsRANF(n) 

V2=  2*0-1 

S*V  1**2  ?**  2 

I*  (S.GE.l)  GO  TO  10 

RN1*V1*  SO RT( <-2.?*AL0G (S)  ) /S) 

X1=EX+PN1*ST0 

RETURN 

END 
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The  signature  tables  used  in  TAC  ASSESSOR  were  assumed  to  be  rep¬ 
resentations  of  some  corresponding  value  functions  evaluated  at 
certain  levels  of  the  input  features.  The  theoretical  mean- 
squared-error  due  to  this  substitution  was  calculated  for  signature 
table  1  and  table  5  in  the  FAC  Target  Importance  Tables.  The 
calculations  were  performed  under  two  different  assumption  concern¬ 
ing  the  assumed  value  functions .  / 
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^Case  1  was  that  the  signature  table  values  resulted  from  evaluatir 
the  assumed  value  function  at  the  given  input  feature  values. 

Case  2  was  that  the  function  was  evaluated  at  some  feature  values 
in  the  given  quantization  level  indicated  by  the  input  feature. 

The  mean  squared  error  was  then  expressed  as  a  percentage  of  the 
assumed  mean  squared  signal.  The  percent  errors  derived  were 
2 0 . 4  8%  and  2  7 . 8 1  o  for  signature  table  1  under  Case  1  and  2 
assumptions,  respectively. 

Next,  an  evaluation  simulation  was  performed  on  three  sets  of 
30  targets  whose  features  were  generated  using  uniform, 
exponential,  and  normal  random  variate  generators  scaled  to 
the  appropriate  feature  limits.  The  major  result  of  this  was 
that  a  significant  number  of  the  targets  were  involved  in  ties 
when  evaluated  using  the  signature  table  method.  The  number  of 
ties  ranged  from  13  out  of  30  up  to  28  out  of  30  targets  involved 
in  evaluatin  ties. 

oFinally,  the  Spearman  rank  correlation  coefficient  was  used  to 
'test  the  null  hypothesis  that  there  was  no  association  between 
the  evaluation  methods. .  The  null  hypothesis  was  rejected  in 
every  case.  Based  on  these  results,  it  is  the  recommendation  of 
this  study  that  formal  application  of  multi  attribute  utility 
assessment  techniqeus  be  applied  to  generate  a  substitute  for 
the  signature  table  evaluation  technique  applied  in  this  case. 
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